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ne of the hottest topics m com- 
puter science these days is data 
compression — the art of shoe- 
homing large amounts of data 
into small spaces. Storage re- 
quirements are growing every 
day. Ten years ago, a lONfB 
hard disk could easily accom- 
modate the storage require- 
ments of dozens of programs. Today, if 
you use Windows and three or four Win- 
dows appUcations, it's hard to get by with 
less than 80MB. New technologies, such 
as real-time video playback and digitized 
sound reproduction, tax the capabilities 
of today's hardware even more, because 
a few seconds of uncompressed video or 
high-resolution sound can tie up several 
megabytes of storage space. Qearly, the 
advent of fast, efficient means of com- 
pressing data on PCs comes not a mo- 
ment too soon. 

Data compression comes in many 
forms. Utilities such as PKZIP compre^ 
data as a standalone operation so that 
you can fit more files on a disk or down- 
load files from bulletin boards in less 
time. Disk compression products such as 
Stacker and DOS 6's new DoubleSpace 
compress and uncompress data in real 
time. You may even benefit from data 
compression without knowing it. Most 
PCX and .GIF graphics files, for exam- 
ple, have compression built in. That's 
why a .PCX or .GIF file is typically much 
smaller than a TIFF file of the same im- 
age. Even Windows .BMP files have an 
option for built-in compression. 

There are two types of data compres- 
sion: lossless and lossy. Lossless data 
compression algorithms ensure that no 
information is lost in the compression/de- 
jompression process; the decompressed 
version is exactly the same as the original. 
Lossy compression algorithms give up 
bit-for-bit accuracy for higher compres- 



sion ratios. Lossless compression finds 

application in utilities such as PKZIP and 
in disk compression software. Lossy com- 
pression is typically used to compress 
video and sound images, where the loss 
of small amounts of data does not reduce 
the perceived overall quality of the out- 
put. Naturally, lossy compression would 
be anathema to a disk compression pro- 
gram, because losing a bit here or there 
could completely alter the meaning of the 
data. 

To the person who's not familiar with 
common compression algorithms, data 
compression may seem more like magic 
tlian sdence. To help you understand 
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DOS 6's DoubleSpace 
make your files smaller 
using RLEy Lempel-Ziv- 

Welch, probabilistic, and 
Sbannon-Fano algorithms. 



how this emerging technology works, 
we'll look at a very simple lossless com- 
pression algorithm known as run-length 
encoding. Then we'll briefly discuss the 
compression algorithms employed by 
PKZIP, Version Li (ever wonder what 
PKZIP is really doing when it says 
"shrinking" or "imploding?") and DOS 
6's DoubleSpace. 

RUN-LENGTH ENCODING Run-length en- 
coding, or RLE, is one of the simplest and 
most often used forms of lossless data 
compression. It works best with files that 
contain many consecutive occurrences of 
the same bit pattern, a characteristic that 
usually applies to bitmapped images. 



Here's a simple example illustrating 

how RLE works. Suppose that you want 
to compress a file that contains the fol- 
lowing 32 bytes of information: 

00 32 46 00 00 00 00 05 
99 02 02 02 01 01 01 01 

01 01 01 01 01 01 01 64 
00 00 00 00 00 00 00 00 

If you examine the file closely, you'll no- 
tice that it contains several "runs" of 
bytes with the values 00 and OL If you 
listed the file 's contents in such a way that 
runs, or groups of repeating bytes, were 
listed on the same line, the file would look 
like this: 



32 

45 

00 00 00 00 
05 

99 

02 02 02 

01 01 01 01 01 01 01 «1 01 01 01 

64 

00 00 00 00 00 00 00 00 

Now imagine what would happen if you 
replaced each line in this listing with 2 
bytes: one identifying the number of 
bytes in the run, and another identifying 
the value of the bytes in the run. Written 
this way, the file would look tlus: 

01 00 

01 32 

01 46 

04 00 

01 05 

01 99 

03 02 

11 01 

01 64 

08 00 
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With this simple foim of encoding, 
you just compressed 32 bytes' worth of 

data into 20 bytes. Knowing the compres- 
sion algorithm, a smart program could 
take the 20 encoded bytes and reproduce 
the original file by reversing the encoding 
process. RLE was incorporated into the 
specification for .PCX and .BMP files be- 
cause long nms of identical characters are 
common in bitmapped images. The 
greater the number and length of repeat- 
ing runs that a file contains, the higher 
the compression ratio that RLE can 
achieve. 

There is a downside, however. In the 
worst case — if a file contained no repeat- 
ing bytes — this simple form of RLE com- 
pression would double the size of the file. 
More efficient forms of RLE exist (forms 
that do not expand every nonrepeating 
byte into 2 bytes), but one must still be 
careful when choosing a compression al- 
gorithm. What's good for one file may not 
be ideal for another. That's why a well- 
written compression program must ana- 
lyze a file before compressing it and must 
dioose the compression method that is 
best suited to it. 

PKZIP PKZIP, probably the world's 
leading file compression and archiving 
utility, chooses from three different com- 
pression methods — shrinking, reducing, 
and imploding — depending on the pat- 
terns of data that a file contains. If none 
of these methods is capable of reducing 
the file size, PKZIP simply copies the file 
unchanged to the ZIP file, a process 
called storing. 

Shrinking uses a variation of the fa- 
mous Lempel-Ziv- Welch (LZW) com- 
pression algorithm. LZW works by build- 
ing a dictionary of phrases (groups of one 
or more characters) firom the input 
stream. When a new phrase is encoun- 
tered, the compression engine checks to 
see if that phrase is recorded in the dic- 
tionary. If not, the phrase is added to the 
dictionary and a token that identifies the 
phrase's position in the dictionary is out- 
put. If the phrase was already recorded, 
then the compressor simply outputs the 
token for the existing phrase. Whereas 
RLE lends itself well to sequences of data 
with repeating single characters, LZW 
works very well on data with repeating 
groups of characters. The English lan- 
guage uses many repeating character pat- 
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terns, so LZW is usually a good choice 
for compressing text files. 

One of the factors that affect the effi- 
ciency of LZW compression is the length 
(in bits) of each output token, which in 
turn determines the maximum size of the 
dictionary. Tokens are typically from 9 to 
15 bits in length. Larger tokens work bet- 
ter for large files, but smaller tokens fre- 
quently produce higher compression ra- 
tios in small files. PKZIP uses a dynamic 
sizing method that begins with 9-bit to- 
kens and increases the token size as the 
size of the dictionary increases, up to a 
maximum of 15 bits per token. This tech- 
nique works as well with small files as it 
does with large files because the final to- 
ken size is a function of the file size. The 
chief difference between a classic imple- 
mentation of LZW and the form that 
PKZIP uses is that when the dictionary 
becomes ML, PKZD' only pwrtMSy clears 

^ well-written 
compression pro-am must 
analyze a file before 
compressing it. 



it of phrases. Many LZW compression 
engines clear the dictionary completely 
and start building phrases again from 

scratch. 

Reducing is actually two algorithms in 
one. First PKZIP compresses repeating 

character sequences using an intricate 
form of RLE. Then it compresses the re- 
sult using a probabilistic compression 
method. Probabilistic compression is 
based on the simple premise that you can 
compress a data stream by using varying 
nimibers of bits to represent different 
characters (or character patterns) if char- 
acters that appear often are assigned the 
fewest number of bits and characters that 
appear infrequently are assigned the 
greatest number of bits. For example, 
suppose a word processing document 
contains 1,000 occurrences of the letter 
E but only 20 occurrences of the letter 
X. Stored as 8-bit ASCII text, the letters 
E and X would account for 1,020 bytes. 
Now suppose that the letter E could be 
represented in just 4 bits, while the letter 



X could be represented with 16. Now E 
and X would account for only 540 bytes— 

500 bytes for the letter E (1,000 charac- 
ters at V2 byte per character) and 40 bytes 
for the letter X (20 characters at 2 bytes 
per character). Clearly this is a more effi-^ 
cient way to store the data, and would re4 
suit in a cooapression ratio of almost So 
percent. | 

This is a very simplistic illustration of 
how probabilistic compression works' 
and it ignores the fact that a probabiUt} 
table must be attached to the compressec 
data, partially offsetting the gains 
PKZIP takes this idea a step further h] 
assigning variable-length bit patterns no 
to single characters, but to pairs of char 
acters. Also, in order to reduce memory 
consumption and the size of the probabil 
ity table, PKZIP limits the table to 32 pat 
terns beginning with a given character 
That's fine for text files, because whil< 
there may be many occurrences of th( , 
characters NE, there will likely be no oc 
currences of. say, QX. The maximun 
length of the probabiUty table is 25( 
(which is the number of characters tha 
can be represented with an 8-bit value 
times 32 (the number of combination; 
supported for each starting character ^ 
bytes, or 8K, plus a few bytes extra fo: 
overhead. In reality, the table is usuall] 
much smaller — especially if the data fil( 
itself is not very large. 

Imploding also employs two discret* 
data compression algorithms. First th( 
data is compressed using a form of Lem 
pel-Ziv (LZ) compression called slidini 
dictionary. Sliding dictionary LZ is simi 
lar to LZW in that it identifies repeating 
phrases by outputting tokens that tell the] 
decompression program where in the file' 
the phrase occurred before. But instead 
of checking the entire file for matching 
phrases, it uses only part of the file — spe- 
cifically, a fixed-size block whose address 
is repeatedly incremented as the file is 
read. Hence the term "sliding diction- 
ary." PKZIP uses a 4K or 8K block size, 
which strikes a reasonable balance be- 
tween compression ratio (the larger the 
block size, the more the data can be com- 
pressed) and compression speed (the 
larger the block size, the more time it 
takes to check for repeating patterns). 
This is a classic form of data compression, 
and one that, like LZW, lends itself well 
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The PHd is a Pocket Hard Disk designed 
to meet the demand of new technology 
with user friendly environment It is 
considered as one of the most 
advanced software application portable 
environment. 

There is a range of four storage 
capacities from 60 to 200MBytes to suit 
a wide range of uses and users of data 
and software. 

The PHd can be easily connected to any 
parallel port on the PC or Notebook. 
The PHd driver automatically finds the 
port and adds the command line to 
recognise the PHd as the next logical 
drive. Your software and files are there 
on screen ready to run. 

Power to the PHd can be supplied 
either via the keyboard using an 
adaptor provided, or with portable 
battery power pack or an optional 
external AC adaptor. 



Available Storage 
Capacities 


60, 80,120, 200 MB 


Average Seek 
Time 


16 msec 


Data Transfer 
Rate V ^ ' 


400KB/sec 


Operating Shock 


lOG's 


Non-Operating 
Shock 


150 G's 


Dimensions 


148min X TSinn X 28Biffl 
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manufacturer 
KT TECHNOLOGY (S) PIE UD 
lOOE fxai Panana Road, KT Building, Sinaapore 0S1 1 

yuk m- anmrn m- fr^Bto mK mmm mm 

distributors 
toiwrgy Resource (USA) Inc 
1S4 Goif W Drive, Longwood, Florida 32779, USA 
%m-8e96257 Fax: (407)-788e708 

UitecOatalink 

IS leHson Road, Markham (Toronto), Ontario, Caimb Ut 1E4 
Tet! (4ie)-415014S Fax: (416)^7SS840 

KTIkchnalogy Europe B.V. 

Hastelweg 268, 5652 CN Eindhoven. The Netherlands 

Tel: 31 40 571555 Fax: 31 40 572077 

CIRCLE 1 95 ON READER SERVICE CARD 



to data streaim wi& repea&if ehmm^&i 
sequences. 

After it is compressed using the sliding 
dictionary method, imploded data is then 
compressed again using Shannon-Fano 
trees. The Shannon-Fano algorithm is a 
form of minimum redundancy coding 
that, like the probabilistic compression 
method described earlier, represents oft- 
repeated characters with fewer bits than 
seldom-used characters. Shannon-Fano 
was one of the very first compression al- 
gorithms, developed in the late 1940s by 
Claude Shannon of Bell Labs and R.M. 
Fano of M.I.T. The term tree comes from 
the method normally used to implement 
Shannon-Fano compression, which in- 
volves counting the occurrences of each 
character and constructing a binary tree 
of Is and Os where characters represent 
"leaves." To determine the numeric code 
for a given character, you begin at the top 
of the tree and follow the branches until 
you reach the specified character. Each 
branch represents oait bit of information. 
Often-used characters appear closer to 
the top of the tree, and hence require 
fewer bits to repv^snt in com|3Tessed 
form. 

In December, PKWare introduced 

PKZIP, Version 2.04g, which replaces 
the three methods of data compression 
used in previous versions — shrinking, re- 
ducing, and imploding — with a single 
method called deflating. Deflating is simi- 
lar to imploding except that it uses a 
larger (32K) dictionary size and a more 
sophisticated form of Shannon-Fano en- 
coding, resultii^ in Mgtar &m^^\on 
ratios. 

The process of uncompressing a file is 
usually far easier — and less CPU-inten- 
sive — than compressing it. If you'd like 
to see how PKZIP flies are unzipped, you 
should check out Michael Mefford's 
PCUNZIP utility, which appeared in PC 
Magazine's March 31, 1992, Utilities col- 
umn. You can download it, source code 
and all, from the Utilities/Tips Forum on 
PC MagNet, archived as PCUNZP.ZIP. 
Better yet, you can use it to imzip com- 
pressed files downloaded from on-line 
services. 

DOUBLESPACE DoubleSpace— the disk 
compression driver built into DOS 6 — 

uses a form of sliding-dictionary LZ com- 
pression similar to the form PKZIP uses 



when it "implodes" a file. DoubleSpace 
uses a 4K sliding dictionary to encode 
often-repeated phrases as tokens that 
consist of an offset from the start of the 
current phrase to an earlier occurrence 
of the same phrase, and a count byte that 
identifies the number of characters in the 
phrase. The greatest difference, of 
course, between DoubleSpace and com- 
pression programs such as PKZIP is that 
DoubleSpace does all its compression 
and decompression behind the scenes, f 
DoubleSpace automatically compresses 
data on its way to a disk and uncom- 
presses it when it is read back. PKZIP, 
by contrast, must be run manually each 
time you want a file or groi^) of files com- 
pressed. 

Borrowing an example fi-om the re- 
viewer's guide that Microsoft distributed 
to members of the press prior to the re- 
lease of MS-DOS 6, the phrase 



tiherain in Spain falls mainly on 



the plain 



which requires 43 bytes, would be en- * 
coded as : 

t:he rain <3 , 3>^<9 , 4>f eills 
m<ll,3>ly o«i<34,4>pl<15,3> 

which requires only 37. Numbers in angle 

brackets represent tokens: the token 
<9,4> means "repeat the 4-character se- 
quence that began 9 characters to the left 
of this token." If you work your way 
through the encoded text from left to 
right replacing tokens with literal text, 
you'll reproduce the original, unencoded 
message. That's exactly what Double- 
Space does when it expands data read 
from a compressed volume. 

FSRTHER READIHG If you're interested 
in learning more about data compression 
methods, there are several books avail- 
able on the subject. My personal favorite 
is The Data Compression Boole, by Mark 
Nelson (M&T Books. 1992). Unlike 
many other books on data compression. 
The Data Compression Book is short on 
equations and long on clear, thoughtfully 
written explanations. It discusses lossy as 
well as lossless compression methods, 
and, for the programmer, it contains 
plenty of source code examples written 
in CD 
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